Computer systems, networks and data centers are exposed to a constant and differing variety of attacks that may compromise the security and/or operation of the system. Examples include various forms of malicious software program attacks such as viruses, worms, Trojan horses and the like that computer systems can obtain over a network such as the Internet. Quite often, users of such computer systems are not even aware that such malicious programs have been obtained within the computer system. Once resident within a computer, a malicious program that executes might disrupt operation of the computer to a point of inoperability and/or might spread itself to other computers within a network or data center by exploiting vulnerabilities of the computer's operating system or resident application programs. Virus attacks, worm attacks, and Trojan horse attacks are variants of each other that generally involve the execution of a program, for which a user often is unaware of its existence, that performs some undesired processing operations to comprise a computer's proper operation.
Other malicious programs operate within a computer to secretly extract and transmit information within the computer to remote computer systems for various suspect purposes. As an example, spyware is a form of software that can execute in the background (e.g., unbeknownst to users) of a computer system and can perform undesirable processing operations such as tracking, recording and transmitting user input from the spyware-resident computer system to a remote computer system. Spyware can allow remote computers to silently obtain otherwise confidential information such as usernames and passwords required to access protected data, lists, contents of files or even a remote web site's user account information.
Computer system developers, software developers and security experts have produced many types of conventional preventive measures that operate within conventional computer systems in an attempt to prevent operation of malicious programs from stealing information or from compromising proper operation of the computer systems. As an example, conventional virus detection software operates to periodically download a set of virus definitions from a remotely located server. Once the virus detection software obtains the definitions, the security software can monitor incoming data received by the computer system, such as email messages containing attachments, to identify viruses defined within the virus definitions that might be present within the data accessed by the computer. Such data might be obtained over a network or might be unknowingly resident on a computer readable medium, such as a disk or CD-ROM, which a user inserts into the computer. Upon detection of inbound data containing a virus or other malicious program, the virus detection software can quarantine the inbound data so that a user of the computer system will not execute code or access the data containing the detected virus that might result in compromising the computer's operation.
Other examples of conventional malicious attacks, intrusions, or undesirable processing that can cause problems within computer systems or even entire computer networks include denial-of-service attacks, buffer overflow operations, execution of malformed application data, and execution of malicious mobile code. A denial-of-service attack operates to provide an intentional simultaneous barrage of packets (e.g., many connection attempts) emanating from many different computer systems to one or more target computer systems, such as a web site, in order to intentionally cause an overload in processing capabilities of the target computer resulting in disruption of service or a business function provided by the target computer. Denial of service attacks may also seek to crash the targeted machine, rather than simply consume resources. Buffer overflow attacks occur when programs do not provide appropriate checks of data stored in internal data structures within the software that result in overwriting of surrounding areas of memory. Attacks based on buffer overflows might allow an attacker to execute arbitrary code on the target system to invoke privileged access, destroy data, or perform other undesirable functions. Malformed application data attacks might result in an application containing a code section that, if executed, provides access to resources that would otherwise be private to the application. Such attacks can expose vulnerabilities due to an incorrect implementation of the application, for example by failing to provide appropriate data validity checks or allowing data stream parsing errors.
Many of the conventional malicious programs and mechanisms for attack of computer systems, such as viruses and worms, include the ability to redistribute themselves to other computer systems or devices within a computer network, such that several computers become infected and experience the malicious processing activities discussed above. Some conventional attempts to prevent redistribution of malicious programs include implementing malicious program detection mechanisms such as virus detection software within firewalls or gateways between different portions of networked computer systems in order to halt propagation of malicious programs to sub-networks.
Another conventional methodology used to control behavior in a system is through the use of an access control list (ACL). The access control list is a concept in computer security, used to enforce privilege separation. It is a means of determining the appropriate access rights to a given object given certain aspects of the user process that is requesting them, principally the process's user identity.
An ACL typically comprises a table that tells a computer operating system which access rights each user has to a particular system object, such as a file directory or individual file. Each object has a security attribute that identifies its access control list. The list has an entry for each system user with access privileges. The most common privileges include the ability to read a file (or all the files in a directory), to write to the file or files, and to execute the file (if it is an executable file, or program). Each ACL has one or more access control entries (ACEs) including the name of a user or group of users. For each of these users or groups roles, the access privileges are stated in a string of bits called an access mask. Generally, the system administrator or the object owner creates the access control list for an object.
Unix-derived systems use a setuid/gid mechanism to change a process's privileges. A user process executing a setuid/gid program gets the “effective” user ID equal to that of the program's owner. This method is commonly used to allow users a kind of privileged access implemented by the setuid/gid program (e.g., changing a user's password). Hence, nefarious users have long targeted setuid/gid files and processes. A common technique used by such nefarious users is to alter a setuid/gid program to allow an intruder to gain unauthorized privileged access, or to subvert a setuid/gid program (e.g., via a buffer overflow) to perform unauthorized operations. Thus, setuid/gid executables (particularly those owned by a superuser), as well as the behavior of processes created from setuid/gid executables, require extra attention and protection.
There have been various intrusion-detection solutions to ensure integrity of setuid files. Some involve running auditing tools that detect any changes in setuid files by examining modification timestamps, checksums and the like. Unix-like operating systems aiming to provide increased security (such as SELinux or GEMSOS) choose to limit the scope of the setuid operation by obviating the all-powerful “root” super-user, or circumscribing setuid with mandatory permissions.
Dynamic behavior of setuid processes is also an object of conventional research and intrusion-detection tools. Some tools analyze audit records from execution of setuid files and detect misuse by spotting statistical anomalies, attack signatures, and other methods.